Method and apparatus for variable accuracy inter-picture timing specification for digital video encoding with reduced requirements for division operations

ABSTRACT

A method and apparatus for performing motion estimation in a digital video system is disclosed. Specifically, the present invention discloses a system that quickly calculates estimated motion vectors in a very efficient manner. In one embodiment, a first multiplicand is determined by multiplying a first display time difference between a first video picture and a second video picture by a power of two scale value. This step scales up a numerator for a ratio. Next, the system determines a scaled ratio by dividing that scaled numerator by a second first display time difference between said second video picture and a third video picture. The scaled ratio is then stored calculating motion vector estimations. By storing the scaled ratio, all the estimated motion vectors can be calculated quickly with good precision since the scaled ratio saves significant bits and reducing the scale is performed by simple shifts.

CLAIM OF BENEFIT OF PRIOR APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.10/792,514, entitled “METHOD AND APPARATUS FOR VARIABLE ACCURACYINTER-PICTURE TIMING SPECIFICATION FOR DIGITAL VIDEO ENCODING WITHREDUCED REQUIREMENTS FOR DIVISION OPERATIONS”, filed Mar. 2, 2004, nowissued as U.S. Pat. No. 7,339,991. U.S. patent application Ser. No.10/792,514 is a continuation application of U.S. patent application Ser.No. 10/313,773, now issued as U.S. Pat. No. 6,728,315, entitled “METHODAND APPARATUS FOR VARIABLE ACCURACY INTER-PICTURE TIMING SPECIFICATIONFOR DIGITAL VIDEO ENCODING WITH REDUCED REQUIREMENTS FOR DIVISIONOPERATIONS”, filed Dec. 6, 2002. U.S. patent application Ser. No.10/313,773 claims priority to U.S. Provisional Application 60/398,625,filed Jul. 24, 2002. U.S. patent application Ser. No. 10/792,514, nowissued as U.S. Pat. No. 7,339,991, U.S. patent application Ser. No.10/313,773, now issued as U.S. Pat. No. 6,728,315, and U.S. ProvisionalPatent Application 60/398,625 are incorporated herein by reference.

CROSS REFERENCE TO RELATED APPLICATIONS

This Application is related to U.S. patent application 10/313,773, filedDec. 6, 2002, now issued as U.S. Pat. No. 6,728,315; U.S. patentapplication 10/792,514, filed Mar. 2, 2004, now issued as U.S. Pat. No.7,339,991; U.S. patent application 11/621,977, filed Jan. 10, 2007, nowpublished as U.S. Publication 2007/0183501; U.S. patent application11/621,979, filed Jan. 10, 2007, now published as U.S. Publication2007/0183502; U.S. patent application 10/291,320, filed Nov. 8, 2002,now issued as U.S. Pat. No. 7,088,776; U.S. patent application10/792,669, filed Mar. 2, 2004, now issued as U.S. Pat. No. 7,292,636;U.S. patent application 11/621,969, filed Jan. 10, 2007, now issued asU.S. Pat. No. 7,903,730; U.S. patent application 11/621,971, filed Jan.10, 2007, now issued as U.S. Pat. No. 7,826,528; and U.S. patentapplication 11/621,974, filed Jan. 10, 2007, now issued as U.S. Pat. No.7,668,240.

FIELD OF THE INVENTION

The present invention relates to the field of multimedia compressionsystems. In particular the present invention discloses methods andsystems for specifying variable accuracy inter-picture timing withreduced requirements for processor intensive division operation.

BACKGROUND OF THE INVENTION

Digital based electronic media formats are finally on the cusp oflargely replacing analog electronic media formats. Digital compact discs(CDs) replaced analog vinyl records long ago. Analog magnetic cassettetapes are becoming increasingly rare. Second and third generationdigital audio systems such as Mini-discs and MP3 (MPEG Audio—layer 3)are now taking market share from the first generation digital audioformat of compact discs.

The video media formats have been slower to move to digital storage anddigital transmission formats than audio media. The reason for thisslower digital adoption has been largely due to the massive amounts ofdigital information required to accurately represent acceptable qualityvideo in digital form and the fast processing capabilities needed toencode compressed video. The massive amounts of digital informationneeded to accurately represent video require very high-capacity digitalstorage systems and high-bandwidth transmission systems.

However, video is now rapidly moving to digital storage and transmissionformats. Faster computer processors, high-density storage systems, andnew efficient compression and encoding algorithms have finally madedigital video transmission and storage practical at consumer pricepoints. The DVD (Digital Versatile Disc), a digital video system, hasbeen one of the fastest selling consumer electronic products in years.DVDs have been rapidly supplanting Video-Cassette Recorders (VCRs) asthe pre-recorded video playback system of choice due to their high videoquality, very high audio quality, convenience, and extra features. Theantiquated analog NTSC (National Television Standards Committee) videotransmission system is currently in the process of being replaced withthe digital ATSC (Advanced Television Standards Committee) videotransmission system.

Computer systems have been using various different digital videoencoding formats for a number of years. Specifically, computer systemshave employed different video coder/decoder methods for compressing andencoding or decompressing and decoding digital video, respectively. Avideo coder/decoder method, in hardware or software implementation, iscommonly referred to as a “CODEC”.

Among the best digital video compression and encoding systems used bycomputer systems have been the digital video systems backed by theMotion Pictures Expert Group commonly known by the acronym MPEG. Thethree most well known and highly used digital video formats from MPEGare known simply as MPEG-1, MPEG-2, and MPEG-4. VideoCDs (VCDs) andearly consumer-grade digital video editing systems use the early MPEG-1digital video encoding format. Digital Versatile Discs (DVDs) and theDish Network brand Direct Broadcast Satellite (DBS) television broadcastsystem use the higher quality MPEG-2 digital video compression andencoding system. The MPEG-4 encoding system is rapidly being adapted bythe latest computer based digital video encoders and associated digitalvideo players.

The MPEG-2 and MPEG-4 standards compress a series of video frames orvideo fields and then encode the compressed frames or fields into adigital bitstream. When encoding a video frame or field with the MPEG-2and MPEG-4 systems, the video frame or field is divided into arectangular grid of pixelblocks. Each pixelblock is independentlycompressed and encoded.

When compressing a video frame or field, the MPEG-4 standard maycompress the frame or field into one of three types of compressed framesor fields: Intra-frames (I-frames), Unidirectional Predicted frames(P-frames), or Bi-Directional Predicted frames (B-frames). Intra-framescompletely independently encode an independent video frame with noreference to other video frames. P-frames define a video frame withreference to a single previously displayed video frame. B-frames definea video frame with reference to both a video frame displayed before thecurrent frame and a video frame to be displayed after the current frame.Due to their efficient usage of redundant video information, P-framesand B-frames generally provide the best compression.

SUMMARY OF THE INVENTION

A method and apparatus for performing motion estimation in a video codesis disclosed. Specifically, the present invention discloses a systemthat quickly calculates estimated motion vectors in a very efficientmanner without requiring an excessive number of division operations.

In one embodiment, a first multiplicand is determined by multiplying afirst display time difference between a first video picture and a secondvideo picture by a power of two scale value. This step scales up anumerator for a ratio. Next, the system determines a scaled ratio bydividing that scaled numerator by a second first display time differencebetween said second video picture and a third video picture. The scaledratio is then stored to be used later for calculating motion vectorestimations. By storing the scaled ratio, all the estimated motionvectors can be calculated quickly with good precision since the scaledratio saves significant bits and reducing the scale is performed bysimple shifts thus eliminating the need for time consuming divisionoperations.

Other objects, features, and advantages of present invention will beapparent from the company drawings and from the following detaileddescription.

BRIEF DESCRIPTION OF THE DRAWINGS

The objects, features, and advantages of the present invention will beapparent to one skilled in the art, in view of the following detaileddescription in which:

FIG. 1 illustrates a high-level block diagram of one possible digitalvideo encoder system.

FIG. 2 illustrates a series of video pictures in the order that thepictures should be displayed wherein the arrows connecting differentpictures indicate inter-picture dependency created using motioncompensation.

FIG. 3 illustrates the video pictures from FIG. 2 listed in a preferredtransmission order of pictures wherein the arrows connecting differentpictures indicate inter-picture dependency created using motioncompensation.

FIG. 4 graphically illustrates a series of video pictures wherein thedistances between video pictures that reference each other are chosen tobe powers of two.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

A method and system for specifying Variable Accuracy Inter-PictureTiming in a multimedia compression and encoding system with reducedrequirements for division operations is disclosed. In the followingdescription, for purposes of explanation, specific nomenclature is setforth to provide a thorough understanding of the present invention.However, it will be apparent to one skilled in the art that thesespecific details are not required in order to practice the presentinvention. For example, the present invention has been described withreference to the MPEG multimedia compression and encoding system.However, the same techniques can easily be applied to other types ofcompression and encoding systems.

Multimedia Compression and Encoding Overview

FIG. 1 illustrates a high-level block diagram of a typical digital videoencoder 100 as is well known in the art. The digital video encoder 100receives an incoming video stream of video frames 105 at the left of theblock diagram. The digital video encoder 100 partitions each video frameinto a grid of pixelblocks. The pixelblocks are individually compressed.Various different sizes of pixelblocks may be used by different videoencoding systems. For example, different pixelblock resolutions include8×8, 8×4, 16×8, 4×4, etc. Furthermore, pixelblocks are occasionallyreferred to as ‘macroblocks.’ This document will use the term pixelblockto refer to any block of pixels of any size.

A Discrete Cosine Transformation (DCT) unit 110 processes eachpixelblock in the video frame. The frame may be processed independently(an intra-frame) or with reference to information from other framesreceived from the motion compensation unit (an inter-frame). Next, aQuantizer (Q) unit 120 quantizes the information from the DiscreteCosine Transformation unit 110. Finally, the quantized video frame isthen encoded with an entropy encoder (H) unit 180 to produce an encodedbitstream. The entropy encoder (H) unit 180 may use a variable lengthcoding (VLC) system.

Since an inter-frame encoded video frame is defined with reference toother nearby video frames, the digital video encoder 100 needs to createa copy of how each decoded frame will appear within a digital videodecoder such that inter-frames may be encoded. Thus, the lower portionof the digital video encoder 100 is actually a digital video decodersystem. Specifically, an inverse quantizer (Q⁻¹) unit 130 reverses thequantization of the video frame information and an inverse DiscreteCosine Transformation (DCT⁻¹) unit 140 reverses the Discrete CosineTransformation of the video frame information. After all the DCTcoefficients are reconstructed from inverse Discrete CosineTransformation (DCT⁻¹) unit 140, the motion compensation unit will usethat information, along with the motion vectors, to reconstruct theencoded video frame. The reconstructed video frame is then used as thereference frame for the motion estimation of the later frames.

The decoded video frame may then be used to encode inter-frames(P-frames or B-frames) that are defined relative to information in thedecoded video frame. Specifically, a motion compensation (MC) unit 150and a motion estimation (ME) unit 160 are used to determine motionvectors and generate differential values used to encode inter-frames.

A rate controller 190 receives information from many differentcomponents in a digital video encoder 100 and uses the information toallocate a bit budget for each video frame. The rate controller 190should allocate the bit budget in a manner that will generate thehighest quality digital video bit stream that that complies with aspecified set of restrictions. Specifically, the rate controller 190attempts to generate the highest quality compressed video stream withoutoverflowing buffers (exceeding the amount of available memory in a videodecoder by sending more information than can be stored) or underflowingbuffers (not sending video frames fast enough such that a video decoderruns out of video frames to display).

Digital Video Encoding with Pixelblocks

In some video signals the time between successive video pictures (framesor fields) may not be constant. (Note: This document will use the termvideo pictures to generically refer to video frames or video fields.)For example, some video pictures may be dropped because of transmissionbandwidth constraints. Furthermore, the video timing may also vary dueto camera irregularity or special effects such as slow motion or fastmotion. In some video streams, the original video source may simply havenon-uniform inter-picture times by design. For example, synthesizedvideo such as computer graphic animations may have non-uniform timingsince no arbitrary video timing is imposed by a uniform timing videocapture system such as a video camera system. A flexible digital videoencoding system should be able to handle non-uniform video picturetiming.

As previously set forth, most digital video encoding systems partitionvideo pictures into a rectangular grid of pixelblocks. Each individualpixelblock in a video picture is independently compressed and encoded.Some video coding standards, e.g., ISO MPEG or ITU H.264, use differenttypes of predicted pixelblocks to encode video pictures. In onescenario, a pixelblock may be one of three types:

-   -   1. I-pixelblock—An Intra (I) pixelblock uses no information from        any other video pictures in its coding (it is completely        self-defined);    -   2. P-pixelblock—A unidirectionally predicted (P) pixelblock        refers to picture information from one preceding video picture;        or    -   3. B-pixelblock—A bi-directional predicted (B) pixelblock uses        information from one preceding picture and one future video        picture.

If all the pixelblocks in a video picture are Intra-pixelblocks, thenthe video picture is an Intra-frame. If a video picture only includesunidirectional predicted macro blocks or intra-pixelblocks, then thevideo picture is known as a P-frame. If the video picture contains anybi-directional predicted pixelblocks, then the video picture is known asa B-frame. For the simplicity, this document will consider the casewhere all pixelblocks within a given picture are of the same type.

An example sequence of video pictures to be encoded might be representedas:

I₁ B₂ B₃ B₄ P₅ B₆ B₇ B₈ B₉ P₁₀ B₁₁ P₁₂ B₁₃ I₁₄ . . .

where the letter (I, P, or B) represents if the video picture is anI-frame, P-frame, or B-frame and the number represents the camera orderof the video picture in the sequence of video pictures. The camera orderis the order in which a camera recorded the video pictures and thus isalso the order in which the video pictures should be displayed (thedisplay order).

The previous example series of video pictures is graphically illustratedin FIG. 2. Referring to FIG. 2, the arrows indicate that pixelblocksfrom a stored picture (I-frame or P-frame in this case) are used in themotion compensated prediction of other pictures.

In the scenario of FIG. 2, no information from other pictures is used inthe encoding of the intra-frame video picture I₁. Video picture P₅ is aP-frame that uses video information from previous video picture I₁ inits coding such that an arrow is drawn from video picture I₁ to videopicture P₅. Video picture B₂, video picture B₃, video picture B₄ all useinformation from both video picture I₁ and video picture P₅ in theircoding such that arrows are drawn from video picture I₁ and videopicture P₅ to video picture B₂, video picture B₃, and video picture B₄.As stated above the inter-picture times are, in general, not the same.

Since B-pictures use information from future pictures (pictures thatwill be displayed later), the transmission order is usually differentthan the display order. Specifically, video pictures that are needed toconstruct other video pictures should be transmitted first. For theabove sequence, the transmission order might be:

I₁ P₅ B₂ B₃ B₄ P₁₀ B₆ B₇ B₈ B₉ P₁₂ B₁₁ I₁₄ B₁₃ . . .

FIG. 3 graphically illustrates the preceding transmission order of thevideo pictures from FIG. 2. Again, the arrows in the figure indicatethat pixelblocks from a stored video picture (I or P in this case) areused in the motion compensated prediction of other video pictures.

Referring to FIG. 3, the system first transmits I-frame I₁ which doesnot depend on any other frame. Next, the system transmits P-frame videopicture P₅ that depends upon video picture I₁. Next, the systemtransmits B-frame video picture B₂ after video picture P₅ even thoughvideo picture B₂ will be displayed before video picture P₅. The reasonfor this is that when it comes time to decode video picture B₂, thedecoder will have already received and stored the information in videopictures I₁ and P₅ necessary to decode video picture B₂. Similarly,video pictures I₁ and P₅ are ready to be used to decode subsequent videopicture B₃ and video picture B₄. The receiver/decoder reorders the videopicture sequence for proper display. In this operation I and P picturesare often referred to as stored pictures.

The coding of the P-frame pictures typically utilizes MotionCompensation, wherein a Motion Vector is computed for each pixelblock inthe picture. Using the computed motion vector, a prediction pixelblock(P-pixelblock) can be formed by translation of pixels in theaforementioned previous picture. The difference between the actualpixelblock in the P-frame picture and the prediction pixelblock is thencoded for transmission.

P-Pictures

The coding of P-Pictures typically utilize Motion Compensation (MC),wherein a Motion Vector (MV) pointing to a location in a previouspicture is computed for each pixelblock in the current picture. Usingthe motion vector, a prediction pixelblock can be formed by translationof pixels in the aforementioned previous picture. The difference betweenthe actual pixelblock in the P-Picture and the prediction pixelblock isthen coded for transmission.

Each motion vector may also be transmitted via predictive coding. Forexample, a motion vector prediction may be formed using nearby motionvectors. In such a case, then the difference between the actual motionvector and the motion vector prediction is coded for transmission.

B-Pictures

Each B-pixelblock uses two motion vectors: a first motion vectorreferencing the aforementioned previous video picture and a secondmotion vector referencing the future video picture. From these twomotion vectors, two prediction pixelblocks are computed. The twopredicted pixelblocks are then combined together, using some function,to form a final predicted pixelblock. As above, the difference betweenthe actual pixelblock in the B-frame picture and the final predictedpixelblock is then encoded for transmission.

As with P-pixelblocks, each motion vector (MV) of a B-pixelblock may betransmitted via predictive coding. Specifically, a predicted motionvector is formed using nearby motion vectors. Then, the differencebetween the actual motion vector and the predicted is coded fortransmission.

However, with B-pixelblocks the opportunity exists for interpolatingmotion vectors from motion vectors in the nearest stored picturepixelblock. Such motion vector interpolation is carried out both in thedigital video encoder and the digital video decoder.

This motion vector interpolation works particularly well on videopictures from a video sequence where a camera is slowly panning across astationary background. In fact, such motion vector interpolation may begood enough to be used alone. Specifically, this means that nodifferential information needs be calculated or transmitted for theseB-pixelblock motion vectors encoded using interpolation.

To illustrate further, in the above scenario let us represent theinter-picture display time between pictures i and j as D_(i,j), i.e., ifthe display times of the pictures are T_(i) and T_(j), respectively,thenD _(i,j) =T _(i) −T _(j) from which it follows thatD _(i,k) =D _(i,j) +D _(j,k)D _(i,k) =−D _(k,i)Note that D_(i,j) may be negative in some cases.

Thus, if MV_(5,1) is a motion vector for a P₅ pixelblock as referencedto I₁, then for the corresponding pixelblocks in B₂, B₃ and B₄ themotion vectors as referenced to I₁ and P₅, respectively would beinterpolated byMV_(2,1)=MV_(5,1) *D _(2,1) /D _(5,1)MV_(5,2)=MV_(5,1) *D _(5,2) /D _(5,1)MV_(3,1)=MV_(5,1) *D _(3,1) /D _(5,1)MV_(5,3)=MV_(5,1) *D _(5,3) /D _(5,1)MV_(4,1)=MV_(5,1) *D _(4,1) /D _(5,1)MV_(5,4)=MV_(5,1) *D _(5,4) /D _(5,1)Note that since ratios of display times are used for motion vectorprediction, absolute display times are not needed. Thus, relativedisplay times may be used for D_(i,j) inter-picture display time values.

This scenario may be generalized, as for example in the H.264 standard.In the generalization, a P or B picture may use any previouslytransmitted picture for its motion vector prediction. Thus, in the abovecase picture B₃ may use picture I₁ and picture B₂ in its prediction.Moreover, motion vectors may be extrapolated, not just interpolated.Thus, in this case we would have:MV_(3,1)=MV_(2,1) *D _(3,1) /D _(2,1)Such motion vector extrapolation (or interpolation) may also be used inthe prediction process for predictive coding of motion vectors.

Encoding Inter-Picture Display Times

The variable inter-picture display times of video sequences should beencoded and transmitted in a manner that renders it possible to obtain avery high coding efficiency and has selectable accuracy such that itmeets the requirements of a video decoder. Ideally, the encoding systemshould simplify the tasks for the decoder such that relatively simplecomputer systems can decode the digital video.

The variable inter-picture display times are potentially needed in anumber of different video encoding systems in order to computedifferential motion vectors, Direct Mode motion vectors, and/or ImplicitB Prediction Block Weighting.

The problem of variable inter-picture display times in video sequencesis intertwined with the use of temporal references. Ideally, thederivation of correct pixel values in the output pictures in a videoCODEC should be independent of the time at which that picture is decodedor displayed. Hence, timing issues and time references should beresolved outside the CODEC layer.

There are both coding-related and systems-related reasons underlying thedesired time independence. In a video CODEC, time references are usedfor two purposes:

(1) To establish an ordering for reference picture selection; and

(2) To interpolate motion vectors between pictures.

To establish an ordering for reference picture selection, one may simplysend a relative position value. For example, the difference between theframe position N in decode order and the frame position M in the displayorder, i.e., N−M. In such an embodiment, time-stamps or other timereferences would not be required. To interpolate motion vectors,temporal distances would be useful if the temporal distances could berelated to the interpolation distance. However, this may not be true ifthe motion is non-linear. Therefore, sending parameters other thantemporal information for motion vector interpolation seems moreappropriate.

In terms of systems, one can expect that a typical video CODEC is partof a larger system where the video CODEC coexists with other video (andaudio) CODECs. In such multi-CODEC systems, good system layering anddesign requires that general functions, which are logicallyCODEC-independent such as timing, be handled by the layer outside theCODEC. The management of timing by the system and not by each CODECindependently is critical to achieving consistent handling of commonfunctions such as synchronization. For instance in systems that handlemore than one stream simultaneously, such as a video/audio presentation,timing adjustments may sometimes be needed within the streams in orderto keep the different streams synchronized. Similarly, in a system thathandles a stream from a remote system with a different clock timingadjustments may be needed to keep synchronization with the remotesystem. Such timing adjustments may be achieved using time stamps. Forexample, time stamps that are linked by means of “Sender Reports” fromthe transmitter and supplied in RTP in the RTP layer for each stream maybe used for synchronization. These sender reports may take the form of:

Video RTP TimeStamp X is aligned with reference timestamp Y

Audio RTP TimeStamp W is aligned with reference timestamp Z

Wherein the wall-clock rate of the reference timestamps is known,allowing the two streams to be aligned. However, these timestampreferences arrive both periodically and separately for the two streams,and they may cause some needed re-alignment of the two streams. This isgenerally achieved by adjusting the video stream to match the audio orvice-versa. System handling of time stamps should not affect the valuesof the pixels being displayed. More generally, system handling oftemporal information should be performed outside the CODEC.A Specific Example

As set forth in the previous section, the problem in the case of nonuniform inter-picture times is to transmit the inter-picture displaytime values D_(i,j) to the digital video receiver in an efficientmanner. One method of accomplishing this goal is to have the systemtransmit the display time difference between the current picture and themost recently transmitted stored picture for each picture after thefirst picture. For error resilience, the transmission could be repeatedseveral times within the picture. For example, the display timedifference may be repeated in the slice headers of the MPEG or H.264standards. If all slice headers are lost, then presumably other picturesthat rely on the lost picture for decoding information cannot be decodedeither.

Thus, with reference to the example of the preceding section, a systemwould transmit the following inter-picture display time values:

D_(5,1) D_(2,5) D_(3,5) D_(4,5) D_(10,5) D_(6,10) D_(7,10) D_(8,10)D_(9,10) D_(12,10) D_(11,12) D_(14,12) D_(13,14) . . .

For the purpose of motion vector estimation, the accuracy requirementsfor the inter-picture display times D_(i,j) may vary from picture topicture. For example, if there is only a single B-frame picture B₆halfway between two P-frame pictures P₅ and P₇, then it suffices to sendonly:D _(7,5)=2 and D _(6,7)=−1where the D_(i,j) inter-picture display time values are relative timevalues.

If, instead, video picture B₆ is only one quarter the distance betweenvideo picture P₅ and video picture P₇ then the appropriate D_(i,j)inter-picture display time values to send would be:D_(7,5)=4 and D_(6,7)=−1Note that in both of the preceding examples, the display time betweenthe video picture B₆ and video picture video picture P₇ (inter-picturedisplay time D_(6,7)) is being used as the display time “unit” value. Inthe most recent example, the display time difference between videopicture P₅ and picture video picture P₇ (inter-picture display timeD_(6,7)) is four display time “units” (4*D_(6,7))Improving Decoding Efficiency

In general, motion vector estimation calculations are greatly simplifiedif divisors are powers of two. This is easily achieved in our embodimentif D_(i,j) (the inter-picture time) between two stored pictures ischosen to be a power of two as graphically illustrated in FIG. 4.Alternatively, the estimation procedure could be defined to truncate orround all divisors to a power of two.

In the case where an inter-picture time is to be a power of two, thenumber of data bits can be reduced if only the integer power (of two) istransmitted instead of the full value of the inter-picture time. FIG. 4graphically illustrates a case wherein the distances between picturesare chosen to be powers of two. In such a case, the D_(3,1) display timevalue of 2 between video picture P₁ and picture video picture P₃ istransmitted as 1 (since 2¹=2) and the D_(7,3) display time value of 4between video picture P₇ and picture video picture P₃ can be transmittedas 2 (since 2²=4).

Alternatively, the motion vector interpolation of extrapolationoperation can be approximated to any desired accuracy by scaling in sucha way that the denominator is a power of two. (With a power of two inthe denominator division may be performed by simply shifting the bits inthe value to be divided.) For example,D_(5,4)/D_(5,1)˜Z_(5,4)/PWhere the value P is a power of two and Z_(5,4)=P*D_(5,4)/D_(5,1) isrounded or truncated to the nearest integer. The value of P may beperiodically transmitted or set as a constant for the system. In oneembodiment, the value of P is set as P=2⁸=256.

The advantage of this approach is that the decoder only needs to computeZ_(5,4) once per picture or in many cases the decoder may pre-computeand store the Z value. This allows the decoder to avoid having to divideby D_(5,1) for every motion vector in the picture such that motionvector interpolation may be done much more efficiently. For example, thenormal motion vector calculation would be:MV_(5,4)=MV_(5,1) *D _(5,4) /D _(5,1)But if we calculate and store Z_(5,4) wherein Z_(5,4)=P*D_(5,4)/D_(5,1)thenMV_(5,4)=MV_(5,1) *Z _(5,4) /PBut since the P value has been chosen to be a power of two, the divisionby P is merely a simple shift of the bits. Thus, only a singlemultiplication and a single shift are required to calculate motionvectors for subsequent pixelblocks once the Z value has been calculatedfor the video picture. Furthermore, the system may keep the accuracyhigh by performing all divisions last such that significant bits are notlost during the calculation. In this manner, the decoder may performexactly the same as the motion vector interpolation as the encoder thusavoiding any mismatch problems that might otherwise arise.

Since division (except for division by powers of two) is a much morecomputationally intensive task for a digital computer system thanaddition or multiplication, this approach can greatly reduce thecomputations required to reconstruct pictures that use motion vectorinterpolation or extrapolation.

In some cases, motion vector interpolation may not be used. However, itis still necessary to transmit the display order of the video picturesto the receiver/player system such that the receiver/player system willdisplay the video pictures in the proper order. In this case, simplesigned integer values for D_(i,j) suffice irrespective of the actualdisplay times. In some applications only the sign (positive or negative)may be needed to reconstruct the picture ordering.

The inter-picture times D_(i,j) may simply be transmitted as simplesigned integer values. However, many methods may be used for encodingthe D_(i,j) values to achieve additional compression. For example, asign bit followed by a variable length coded magnitude is relativelyeasy to implement and provides coding efficiency.

One such variable length coding system that may be used is known as UVLC(Universal Variable Length Code). The UVLC variable length coding systemis given by the code words:

$\begin{matrix}{1 =} & \; & \; & \; & 1 & \; & \; & \; \\{2 =} & \; & \; & 0 & 1 & 0 & \; & \; \\{3 =} & \; & \; & 0 & 1 & 1 & \; & \; \\{4 =} & \; & 0 & 0 & 1 & 0 & 0 & \; \\{5 =} & \; & 0 & 0 & 1 & 0 & 1 & \; \\{6 =} & \; & 0 & 0 & 1 & 1 & 0 & \; \\{7 =} & \; & 0 & 0 & 1 & 1 & 1 & \; \\{8 =} & 0 & 0 & 0 & 1 & 0 & 0 & {0\mspace{11mu}\ldots}\end{matrix}$

Another method of encoding the inter-picture times may be to usearithmetic coding. Typically, arithmetic coding utilizes conditionalprobabilities to effect a very high compression of the data bits.

Thus, the present invention introduces a simple but powerful method ofencoding and transmitting inter-picture display times and methods fordecoding those inter-picture display times for use in motion vectorestimation. The encoding of inter-picture display times can be made veryefficient by using variable length coding or arithmetic coding.Furthermore, a desired accuracy can be chosen to meet the needs of thevideo codec, but no more.

The foregoing has described a system for specifying variable accuracyinter-picture timing in a multimedia compression and encoding system. Itis contemplated that changes and modifications may be made by one ofordinary skill in the art, to the materials and arrangements of elementsof the present invention without departing from the scope of theinvention.

1. A non-transitory computer readable medium storing a computer programfor encoding a plurality of video pictures that comprise first andsecond video pictures, the computer program executable by one or moreprocessing units, the computer program comprising sets of instructionsfor: specifying at least one display order value for computing a firstmotion vector for the first video picture, wherein said computed firstmotion vector for the first video picture is based on a scaled ratio anda second motion vector for the second video picture; and encoding theplurality of video pictures and said display order value into abitstream.
 2. The non-transitory computer readable medium of claim 1,wherein the scaled ratio is based on a ratio between a first displayorder difference value and a second display order difference value. 3.The non-transitory computer readable medium of claim 1, wherein saidorder value is for specifying a position of at least one video picturein a sequence of video pictures.
 4. The non-transitory computer readablemedium of claim 3, wherein the sequence is for displaying the pluralityof video pictures.
 5. The non-transitory computer readable medium ofclaim 1, wherein said order value is compressed in the bitstream byusing variable length encoding.
 6. An encoder for encoding a pluralityof video pictures comprising first and second video pictures, theencoder comprising: a module that specifies at least one order value forcomputing a first motion vector for the first video picture, whereinsaid computed first motion vector for the first video picture is basedon a scaled ratio and a second motion vector for the second videopicture; and a module that encodes the plurality of video pictures andsaid order value into a bitstream.
 7. The encoder of claim 6, whereinthe scaled ratio is based on a ratio between a first order differencevalue and a second order difference value.
 8. The encoder of claim 6,wherein said order value is for specifying a position of at least onevideo picture in a sequence of video pictures.
 9. The encoder of claim8, wherein the sequence is for displaying the plurality of videopictures.
 10. The encoder of claim 6, wherein said order value iscompressed in the bitstream by using variable length encoding.
 11. Amethod of encoding a plurality of video pictures that comprise first andsecond video pictures, the method comprising: specifying at least oneorder value for computing a first motion vector for the first videopicture, wherein said computed first motion vector for the first videopicture is based on a scaled ratio and a second motion vector for thesecond video picture; and encoding the plurality of video pictures andsaid order value into a bitstream.
 12. The method of claim 11, whereinthe scaled ratio is based on a ratio between a first order differencevalue and a second order difference value.
 13. The method of claim 11,wherein said order value is for specifying a position of at least onevideo picture in a sequence of video pictures.
 14. The method of claim13, wherein the sequence is for displaying the plurality of videopictures.
 15. The method of claim 11, wherein said order value iscompressed in the bitstream by using variable length encoding.